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Congjun  Wu,  Department  of  Physies,  University  of  California,  San  Diego 

We  present  our  researeh  for  the  ARO  award  W91  INFOS  10291  with  the  title  ‘Wove/  quantum 
states  with  exotic  spin  properties  -  Unconventional  generalizations  of  magnetism""  in  the  entire 
supporting  period  (Jul.  2008-  Sept.  2011).  My  group  has  made  signifieant  progress  in  several 
directions  related  to  the  proposed  results.  We  present  our  research  on  the  microscopic  mechanism 
on  the  nematic  states  observed  in  Sr3Ru207  in  Sect.  1.  The  study  of  strong  correlation  effects  in 
topological  insulators  is  presented  in  Sect.  2.  The  study  of  a  novel  topological  insulators  based  on 
high  dimensional  Landau  levels  is  presented  in  Sect.  3.  In  Sect.  4,  we  present  our  study  on  the 
iron-based  superconductivity.  The  summary  of  important  results  in  given  in  Sect.  5. 

Under  the  support  by  ARO,  we  have  an  excellent  publication  record  in  high  profile  research 
journals.  We  have  published  one  article  in  Science,  four  articles  in  Physical  Review  Letters,  four 
articles  in  Physical  Review  B,  and  three  papers  under  review  at  Physical  Review. 

Below  we  will  describe  the  progress  in  each  direction. 

1  Unconventional  magnetism  in  Sr3Ru207 

We  extended  our  idea  of  unconventional  magnetism  of  the  Fermi  liquid  theory  [1,  2]  to  orbital 
band  systems,  and  applied  it  for  the  nematic  metamagnetic  state  observed  in  the  strontium  ruthen- 
ate  material  Sr3Ru207  in  A.  Mackenzie’s  group.  The  microscopic  theory  for  the  nematic  metam¬ 
agnetic  transition  was  constructed  [3].  This  state  is  essentially  the  unconventional  magnetic  states 
we  proposed  in  the  J-wave  channel  with  a  partial  spin  polarization  [1,2].  In  the  realistic  orbital 
band  systems  of  Sr3Ru207,  it  also  induces  orbital  ordering.  Furthermore,  we  performed  the  cal¬ 
culation  to  explain  the  experimentally  observed  the  quasi-particle  interference  pattern  of  the  STM 
spectroscopy  in  Sr3Ru207  measured  by  J.  C.  Davis’s  group  [4,  5].  We  also  investigated  the  ther¬ 
modynamic  properties  in  the  nematic  regime  [6],  which  was  measured  in  A.  Mackenzie’s  group 
[7,  8]. 

1.1  The  key  questions  of  the  nematic  states  in  Sr3Ru207 

In  the  past  decade,  the  nematic  metamagnetic  states  observed  in  the  external  magnetic  field  have 
aroused  a  great  deal  of  research  interest  in  the  condensed  matter  community.  When  the  B-field 
perpendicular  is  applied  to  the  ab-p\an&  between  7.8  and  8.1  Tesla,  the  transport  properties  exhibit 
a  strong  spontaneous  in-plane  anisotropy  along  the  a  and  b-axis  [9,  10,  11,  12,  13,  14,  15,  16]. 

This  effect  is  of  electronic  origin  because  the  lattice  distortion  observed  is  negligible.  It  has  been 
interpreted  as  a  nematic  state  with  the  anisotropic  distortion  of  the  Fermi  surface  of  the  majority 
spin  polarized  by  the  external  magnetic  field  in  Ref.  [10].  Two  key  questions  have  still  not  been 
answered  satisfyingly  in  previous  theory  work.  First,  Sr3Ru207  is  a  U^-band  (  dxz,  dy^,  and  dxy) 
system  with  the  active  4J-orbitals  of  Ru.  The  dxz  J^^-orbitals  extend  in  the  xz  and  yz  plane, 
respectively,  thus  they  behave  quasi-one  dimensional  in  the  ab-plane.  The  J;^y-orbital  lies  inside  the 
ab-pXawQ,  and  its  band  structure  is  two-dimensional.  Which  bands  are  responsible  for  the  nematic 
behavior?  Second,  the  nematic  metamagnetic  states  require  strong  exchange  interactions  in  the 
J-wave  channel,  while  the  usual  exchange  interaction  from  Coulomb  repulsion  is  dominate  in  the 
5-wave  channel.  How  to  reconcile  this  discrepancy? 


1 


0.08 


Figure  1:  A)  The  hybridized  quasi- ID  bonding  bands  of  dxz  and  dyy.  As  moving  around  the  large  Fermi 
surface,  the  orbital  configuration  exhibits  a  d-wave  pattern,  varying  from  dx^  — >  dy^  — >  dx^  — >  dy^.  B)  An 
idealized  Fermi  surface  by  rounding  off  the  large  Fermi  surface  in  A).  C)  Self-consistent  calculation  results 
of  the  spin  magnetization  m  (the  dotted  line)  and  the  the  nematic  order  parameter  n  (the  solid  line)  versus  the 
external  magnetic  field  h.  The  nemafic  ordering  develops  befween  fwo  consecutive  mefamagnefic  fransifions. 
The  fopologies  of  Fermi  surface  in  each  phases  are  skefched  in  fhe  insecfs,  where  fhe  solid  and  dashed  lines 
represenf  fhe  Fermi  surface  of  fhe  majorify  and  fhe  minorify  spin  bands. 

We  have  provided  a  natural  answer  by  extending  our  previous  theory  of  uneonventional  mag¬ 
netism  into  multi-band  systems  in  Ref.  [3].  We  point  out  that  it  is  the  hybridized  quasi-one- 
dimensional  dxz  and  dyz  bands  that  are  responsible  for  the  nematie  ordering  based  on  the  following 
reasoning.  The  nematie  metamagnetie  state  is  only  observed  in  the  bilayer  eompound  Sr3Ru207, 
but  not  in  the  monolayer  eompound  Sr2Ru04,  both  of  whieh  are  the  f2g-orbital  systems.  The  key 
differenee  in  eleetronie  struetures  between  them  is  the  bilayer  splitting,  whieh  is  prominent  for  the 
quasi-one  dimensional  bands  of  dxz  and  dyz  but  small  for  the  two-dimensional  bands  of  dxy  It  is 
natural  to  expeet  that  the  spontaneous  nematie  behavior  oeeurs  in  the  bands  of  dxz  and  dyz  and  is 
aeeompanied  by  an  orbital  ordering.  Furthermore,  our  meehanism  based  on  the  quasi-one  dimen¬ 
sional  band  naturally  generates  the  exehange  interaetion  in  the  J-wave  ehannel.  The  orbital  band 
hybridization  between  dxz  dyz  shifts  a  signifieant  speetra  weight  of  the  exehange  interaetion  into 
the  J-wave  ehannel,  thus  the  nematie  ordering  ean  arise  from  the  eonventional  multi-band  Hubbard 
interaetions. 

1.2  Our  mechanism  -  unconventional  magnetism  in  the  dxz  and  J^^-hands 

We  present  below  a  heuristie  pieture  to  illustrate  how  orbital  hybridization  between  the  dxz  and 
dyz  band  enhanees  the  d-wave  ehannel  exehange  interaetion.  We  first  review  the  exehange  interae¬ 
tion  in  the  usual  single  band  system  in  the  Landau  Fermi  liquid  theory.  At  the  Hartree-Foek  level 
it  is  expressed  as  /"{pi-iPi)  =  —jy{\Pi—P2\),  where  a  represents  the  spin  ehannel  interaetion, 
V{p)  is  the  Fourier  transform  of  the  two-body  interaetion  V{\r\  —r2\),  say,  the  Coulomb  interae¬ 
tion.  The  high  partial  wave  ehannel  eomponents  of  V  (q)  are  usually  weak,  thus  the  eondition  for 
uneonventional  magnetism  is  more  stringent  than  the  usual  the  5-wave  ehannel  ferromagnetism. 

This  situation  is  signifieantly  ehanged  in  the  hybridized  dxz  dyz,  quasi-one  dimensional  bands 
presented  above.  Without  loss  of  generality,  let  us  round  off  the  large  Fermi  surfaee  depieted  in 
Fig.  2  A  to  a  eirele  as  in  Fig.  2  B  assuming  that  the  single  partiele  orbital  eonfiguration  takes 
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the  form  of  |\|/o(i?))  =  e‘P'''u{p)  ®Xa,  where  u{p)  =  (eos(|)p|  +  sin(|)p|  Jy^))  is  the  Bloeh  wave 

funetion  with  the  internal  orbital  eonfigurations;  <^p  is  the  azimuthal  angle  of  p;  Xa{<3  =T7i) 
the  spin  eigenstates.  The  exehange  interaetion  between  eleetrons  with  the  same  spin  is  —V {pi  — 
P2)\{u{pi)\u{p2))\'^  —  —V{pi  —  p2)cos^((|)pj  —  (j)p2)-  Beeause  of  this  faetor  of  the  inner  produet, 
the  spin  ehannel  exehange  interaetion  ehanges  into 

fiPhPi)  =  -^[i+cos2(4)pj -^P2)]y(pi -P2)-  (1) 

Therefore  even  if  the  bare  interaetion  V{p\—p2)  is  dominated  by  the  5- wave  ehannel,  the  extra 
J-wave  faetor  arising  from  the  orbital  hybridization  shifts  a  signifieant  weight  into  the  J-wave 
ehannel. 

With  the  above  non-trivial  band  strueture,  we  aehieve  the  experimentally  observed  nematie 
metamagnetie  state  with  the  multi-band  Hubbard  interaetions.  These  interaetions,  ineluding  the 
intra-orbital  and  inter-orbital  repulsions  and  Hunds  rule  eouplings,  are  natural  interaetions  widely 
used  to  deseribe  transition  metal  oxides.  Thus  we  do  not  need  artificial  interactions  as  in  Ref. 
[17,  18,  19].  The  Hubbard  interactions  by  themselves  do  not  result  in  the  exotic  nematic  states 
because  their  contribution  is  concentrated  in  the  isotropic  5-wave  channel  if  there  were  no  band 
hybridization.  However,  when  projected  into  the  hybridized  dxz  and  dy^  bands,  Hubbard  interac¬ 
tions  acquire  the  J-wave  channel  component  with  a  significant  weight  which  is  responsible  for  the 
nematic  states.  The  phase  diagram  from  the  self-consistent  calculation  is  presented  in  Fig.  2  C. 

1.3  Spectroscopic  imaging  scanning  tunneling  microscopy  (STM)  as  a  probe  to  the  nematic 
metamagnetie  state 

Quasi-particle  interference  (QPI)  of  the  spectroscopic  imaging  STM  has  become  a  powerful  method 
to  investigate  electronic  structure  and  competing  orders  in  strongly  correlated  systems  [20,  21,  22, 
23].  It  is  amazing  that  the  real  space  local  probe  of  STM  can  give  rise  to  the  electronic  information 
of  momentum  space.  It  has  the  advantages  of  a  high  energy  resolution  and  the  ability  to  access 
unoccupied  states  above  the  Fermi  energy.  The  process  of  the  QPI  analysis  is  as  follows.  First, 
the  real  space  distribution  of  the  local  density  of  states  around  a  single  impurity  is  measured  by 
STM  with  a  high  precision.  Then  its  Fourier  transform  is  performed,  which  reveals  the  momentum 
change  during  electron  scattering  processes.  Positions  with  high  spectra  weight  correspond  to 
scattering  wavevectors  connecting  regimes  with  high  density  of  states  in  the  iso-energy  line  in 
momentum  space. 

In  the  presence  of  nematic  ordering,  the  Fermi  surface  exhibits  anisotropic  distortion.  The  four¬ 
fold  rotational  symmetry  of  the  Fermi  surface  is  broken  into  two-fold.  This  symmetry  breaking 
nicely  exhibits  in  the  QPI  pattern  in  which  the  stripes  only  extend  along  one  particular  direction 
as  expected  for  a  nematic  order.  Thus  QPI  provides  a  sensitive  probe  to  the  nematic  ordering.  In 
fact,  this  pattern  has  actually  been  seen  in  QPI  spectra  of  the  iron-pnictide  compound  CaFe2As2 
in  the  ai.2-Fermi  surfaces  around  the  F-point  by  the  same  group  of  J.  C.  Davis  [24].  These  Fermi 
surfaces  are  of  the  nature  of  dxz  and  dyz,  thus  our  above  analysis  also  applies. 

Recently,  this  technique  has  been  applied  to  Sr3Ru207  with  1%  Ru  sites  substituted  by  Ti  in 
Davis’  group  [25].  The  Ti  sites  behave  as  scattering  centers.  The  most  prominent  feature  in  their 
results  is  the  characteristic  square-box-like  pattern.  In  our  theory,  this  pattern  is  the  key  feature  of 
the  scattering  from  the  nearly  nested  quasi  one-dimensional  bands  of  dxz  and  dyz  [4].  Our  work 
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Figure  2:  The  Fermi  surfaces  of  Sr3Ru207  using  the  bilayer  tight-binding  model  with  different  interlayer 
bias  Vbias  for  (a)  Vbias/fo  =  0,  (b)  Vbias/fo  =  0.05,  (c)  Vbias/fo  =  0.1,  and  (d)  Vbias/fo  =  0.15.  The  thick  dashed 
lines  mark  the  boundary  of  half  Brillouin  zone  due  to  the  unit  cell  doubling  induced  by  the  rotation  of 
octahedra  oxygens.  Fig.  (a)  represents  the  bulk  Fermi  surfaces  without  interlayer  bias.  The  bonding  {k^  =  0, 
black  solid  lines)  and  the  anti-bonding  bands  {k^  =  n,  red  dashed  lines)  could  cross  since  k^  is  a  good  quantum 
number.  Fig.  (c)  fits  the  Fermi  surfaces  measured  by  ARPES  best.  Fermi  surface  sheets  of  OCi,  a2,  Yi,  Y2^ 
and  p  are  marked.  From  Ref.  [27]. 


provides  a  theoretical  support  of  a  new  detection  method  to  the  nematic  metamagnetic  state  and 
orbital  ordering. 

The  realistic  band  structure  of  Sr3Ru207  We  first  constructed  a  tight-binding  model  to  capture  the 
complicated  Fermi  surface  structures  of  the  bilayer  ruthenate  compound  of  Sr3Ru207,  including  its 
t2g-orbital  structure,  the  spin-orbit  coupling,  the  bilayer  splitting,  and  the  staggered  rotations  of  the 
RuO  octahedra.  The  band  parameters  are  chosen  by  fitting  the  calculated  Fermi  surfaces  with  those 
measured  in  the  ARPES  experiment  performed  by  Shen’s  group  [26].  We  found  that  in  order  to 
explain  the  measured  Fermi  surface  structures,  an  interlayer  bias  potential  Vhias  needs  to  be  added 
between  two  RuO  layers.  This  is  because  two  top  layers  on  the  surface,  which  are  sensitive  to  the 
ARPES  measurement,  are  non-equivalent  with  different  work  functions.  Such  an  effect  is  absent 
in  the  bulk  due  to  the  reflection  symmetry  between  bilayers,  which  corresponds  to  Vbias  =  0-  The 
Eermi  surface  structures  of  Sr3Ru207  with  different  values  of  the  interlayer  bias  Vuas  are  presented 
in  Eig.  2.  As  can  be  clearly  seen,  they  are  very  complicated.  Eig.  2  (c)  is  the  best  fit  to  the  Eermi 
surfaces  of  the  surface  bilayer  measured  in  ARPES.  Eermi  surface  sheets  marked  with  tti,  a2,  Yf 
Y2,  and  P  are  observed  in  ARPES  [26].  The  Yi  sheet  has  dominant  2-D  dxy  orbital  character;  the 
small  Y2  sheets  are  hybridized  between  the  ^i?;cy-orbital  and  the  quasi- ID  bands;  the  ai,2 

sheets  are  mostly  formed  by  quasi- Id  dyz,xz  orbitals. 
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Figure  3:  (LHS)  QPI  imagings  at  different  values  of  bias  (a)  E  =  0,  (b)  E  =  —0.03,  (c)  E  =  —0.06,  and 
(d)  E  =  —0.1.  The  main  features  are  similar  because  the  Fermi  surfaces  of  ai,2  are  relatively  insensitive 
to  E  throughout  this  energy  range.  (RHS)  The  QPI  imagings  with  the  nematic  order.  The  breaking  of  the 
four-fold  symmetry  to  the  two-fold  symmetry  is  clearly  seen.  From  Ref.  [27]. 


The  QPI  spectra  Based  on  the  above  band  structure,  we  perform  the  T -matrix  calculation  for  the 
QPI  spectra  with  different  values  of  the  STM  bias  as  presented  in  Fig.  3  (LHS).  Our  calculation 
exhibits  the  hollow  square-box-like  feature  which  agrees  well  with  the  experimental  observation 
[25].  This  dispersion  of  the  QPI  spectra  with  varying  the  STM  bias  is  also  qualitatively  consistent 
with  the  experimental  results.  This  square-box-like  feature  arises  from  the  scattering  from  the 
nearly  nested  quasi-one-dimensional  bands  of  the  tti  2-  The  brightest  spots  at  the  corners  come 
from  the  quasi- ID  components  of  the  small  Y2-bands.  These  bands  are  close  to  the  van  Hove 
singularity,  and  thus  their  scatterings  are  enhanced. 

QPI  as  a  probe  to  the  nematic  metamagnetic  state  The  experiment  performed  by  Davis’  group  on 
Sr3Ru207  so  far  is  in  the  zero  magnetic  field,  which  is  out  of  the  nematic  phase,  thus  no  4-fold 
symmetry  breaking  effects  were  observed.  In  a  previous  paper  [5],  we  have  shown  that  the  QPI  is 
a  sensitive  method  to  detect  the  nematic  ordering,  which  exhibits  as  the  4-fold  symmetry  breaking 
into  the  2-fold  symmetry.  In  fact,  this  pattern  has  actually  been  seen  in  the  QPI  spectra  of  the  iron- 
pnictide  compound  CaFe2As2  in  the  tti  2-Fermi  surfaces  around  the  F-point  in  the  same  experiment 
group  [28]. 

In  our  new  paper  [27],  we  calculated  the  QPI  pattern  in  the  nematic  state  in  Sr3Ru207  as  de¬ 
picted  in  Fig.  3  (RHS).  In  spite  of  the  complicated  band  structures,  the  symmetry  breaking  pattern 
from  the  4-fold  to  2-fold  is  clearly  seen.  This  feature  can  be  tested  in  future  experiments. 

1.4  Thermodynamic  properties  of  the  unconventional  metamagnetic  states  of  Sr3Ru207 

Rost  et.  al.  [7,  8]  measured  the  entropy  and  specific  heat  near  the  metamagnetic  transition  and  found 
divergences  near  as  C/T  =  A[(//  — 7/^)///^]  with  the  exponent  a  =  1.  This  feature  has  been 
interpreted  as  the  signature  of  strong  correlation  physics  of  quantum  criticality.  Another  puzzling 
is  that  the  boundary  of  the  nematic  regime  in  the  T  —  H  phase  diagram  has  the  “muffin”  shape,  i.e., 
increasing  temperatures  can  drive  the  transition  from  the  isotropic  phase  to  the  nematic  phase.  This 
means  that  the  nematic  phase  has  higher  entropy  than  the  isotropic  phase  at  low  temperatures. 
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Figure  4:  (LHS)  The  entropy  landscape  represented  by  the  quantity  S{B) /T  in  unit  of  k^/ti  within  the  range 
of  0.0045  <  <  0.007.  A  sudden  increase  near  the  nematic  region  (green  area)  is  clearly  seen.  (Middle) 

Phase  diagram  at  finite  temperatures.  Magnitudes  of  the  nematic  ordering  are  represented  by  the  color  scales. 
The  re-entry  of  the  nematic  order  at  higher  temperature  is  seen  at  fields  befween  0.0058  <  <  0.0063. 

(RHS)  The  fofal  DOS  Pro((ro)  as  a  funclion  of  Pfo/(tt>)  does  nol  follow  fhe  rigid  band  picfure  and 
P/of  (ro  =  0)  has  a  sudden  increase  near  fhe  nematic  region.  Insef :  fhe  fofal  DOS  af  zero  field  for  a  wider  range 
of  |co|/ti  <  0.4.  The  peaks  corresponding  fo  van  Hove  singularities  are  near  co/o  Ri  —0.2  and  co/fi  «  0.35. 


Motivated  by  the  observation  that  at  low  temperatures,  both  the  nematic  and  isotropic  phases 
are  Fermi  liquids  from  the  resistivity-temperature  measurements,  we  feel  that  the  self-consistent 
mean-field  theory  (MFT)  is  sufficient  to  capture  the  essential  physics.  In  our  new  paper  [6],  we 
performed  the  self-consistent  MFT  calculations  for  the  thermodynamic  quantities  by  taking  into 
account  the  competition  between  ferromagnetic  and  nematic  orderings,  which  qualitatively  agree 
with  the  experimental  results. 

Entropy  and  DOS  near  the  nematic  region  The  entropy  v.s.  the  Zeeman  energy  are  plotted  in 
Fig.  4  (LHS).  The  increase  of  the  entropy  towards  the  nematic  regime  and  its  non-monotonic 
behavior  in  the  nematic  regime  are  in  agreement  with  experiments,  which  are  combined  effects  of 
van  Hove  singularity  of  DOS  and  interaction.  The  calculated  H  —  T  phase  diagram  is  plotted  in 
Fig.  4  (Middle),  the  shape  of  the  phase  boundary  on  the  high-field  side  agrees  with  the  experiment 
results,  which  means  that  increasing  temperatures  can  drive  the  system  into  the  nematic  phase. 
This  results  are  consistent  with  the  entropy  plotted  in  Fig.  4  (LHS),  which  shows  that  the  entropy 
is  larger  in  the  nematic  phase  than  the  isotropy  phase  at  the  boundary  of  high  field  side.  However, 
our  calculation  for  the  phase  boundary  of  the  low-field  side  does  not  agree  with  experiments. 

Furthermore,  we  also  calculated  the  density  of  states  v.i.  Zeeman  energy,  which  shows  that  the 
increase  of  density  of  states  when  the  field  is  in  the  nematic  regime  as  presented  in  Fig.  4  (RHS). 
The  density  of  states  cannot  be  described  as  a  rigid  Zeeman  shift  in  agreement  with  experiments. 
This  is  a  result  of  spin-orbit  coupling  effect. 

In-plane  B  field  induced  anisotropy  Another  puzzle  remains  to  be  explained  is  the  correlation 
between  transport  anisotropy  and  the  in-plane  component  of  the  B-field  measured  in  experiments 
[11].  Suppose  with  an  in-plane  B-field  along  the  x-axis,  the  resistivity  measurements  show  that  the 
easy  axis  is  along  the  y-direction.  As  pointed  out  by  Raghu  et  al.  [29],  an  uniform  nematic  state 
actually  does  not  exhibit  much  anisotropy  in  the  charge  transport.  The  point  is  that  the  nematic 
phase  is  mostly  associated  with  the  states  near  the  van  Hove  singularity  whose  Fermi  velocities  are 
too  small  to  have  sizable  contribution  to  the  transport  properties.  Therefore  the  observed  anisotropic 
resistivity  is  mostly  likely  due  to  the  scatterings  between  nematic  domains.  However,  it  is  still  not 
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Figure  5 :  Illustration  of  the  energetically-favored  domain  structures  as  the  in-plane  magnetic  field  is  along  x 
axis.  When  the  in-plane  magnetic  field  is  weak,  the  phase  with  the  preferred  dy^-oxhitsA  are  dominant  (shaded 
areas)  and  high-energy  domains  with  the  dxz-oxhitaA  (green  ovals)  could  exist,  (b)  The  domain  walls  extend 
longer  in  y  direction  because  it  costs  less  energies  if  less  y-  than  x-  bonds  are  broken  in  the  J^^-orbital  ordered 
background.  The  white  oval  represents  the  wave  function  of  dyz  orbital  on  each  site.  The  current  flows  much 
more  easily  along  y-axis  than  x-axis  since  the  electrons  suffer  less  domain  scatterings  hopping  along  this 
direction.  As  the  in-plane  magnetic  field  is  strong,  these  high-energy  domains  vanish,  thus  there  is  no  longer 
an  easy  axis  for  the  current  flow. 


clear  why  the  easy  axis  of  eharge  transport  is  perpendieular  to  the  in-plane  5-field. 

We  provide  a  natural  explanation  below  based  on  the  anisotropie  spatial  extension  of  the  domain 
walls  as  explained  below  as  depleted  in  Fig.  5.  Assuming  the  5-field  lying  in  the  xz-plane,  the  in¬ 
plane  orbital  Zeeman  energy  reads  plane  =  —IJBBsmd'^i  aLx,ia,  whieh  eouples  the  dxy  and 
ti?xz-orbitals  and  breaks  the  degeneraey  between  the  dxz  and  Jyj. -orbitals.  Sinee  the  dxy-oxhita\  has 
lower  on-site  energy  due  to  the  erystal  field  splitting  than  that  of  dxz,  the  Jxz-orbital  bands  are  pushed 
to  higher  energy  than  dyz-oxhiidX  bands  by  this  extra  eoupling.  As  a  result,  the  nematie  state  with 
preferred  J^^-orbitals  has  lower  energy  in  the  homogeneous  system.  At  small  angles  of  0,  domains 
with  preferred  ^t^z-orbitals  eould  form  as  depleted  in  Fig.  5  (a)  as  meta-stable  states,  whieh  oeeupy 
less  volume  than  the  majority  domains.  Let  us  eonsider  the  shape  of  the  domain  walls.  Beeause  of 
the  quasi- ID  features,  the  horizontal  (vertieal)  domain  wall  breaks  the  bonds  of  the  -orbital 

as  depleted  in  Fig.  5  (b),  respeetively.  Sinee  the  Jy^-orbital  is  preferred  by  Hin-piam,  the  horizontal 
domain  wall  eosts  more  energy.  Consequently,  the  domain  strueture  illustrated  in  Fig.  5  (a)  with 
longer  vertieal  walls  than  the  horizontal  walls  is  energetieally  favored,  and  eauses  the  easy  axis  of 
y  for  the  eurrent  flow. 

2  Strong  interaction  effects  in  the  Kane-Mele  model  of  topological  insula¬ 
tors 

The  study  of  topologieal  insulators  (TI)  in  both  two  dimensions  (2D)  and  three  dimensions  (3D) 
has  beeome  a  major  foeus  of  eondensed  matter  physies  [30,  31,  32,  33,  34,  35,  36,  37,  38,  39, 
40,  41,  42].  The  quantum  Hall  systems  are  the  prototype  of  topologieal  insulators,  whieh  breaks 
time-reversal  (TR)  symmetry  and  whose  magnetie  band  strueture  is  eharaeterized  by  the  integer 
Chem  number.  In  eontrast,  these  new  TIs  maintain  TR  symmetry,  and  their  band  struetures  are 
eharaeterized  by  the  nontrivial  Z2-index  for  both  2D  and  3D  systems.  On  the  edge  of  the  2D 
topologieal  insulators,  odd  number  ehannels  of  helieal  edge  modes  appear.  On  the  surfaee  of  the  3D 
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topological  insulators,  odd  number  of  surface  Dirac  cones  appear.  Both  of  them  are  stable  against 
disorder  scatterings.  However,  interaction  effects  in  topological  insulators  have  not  been  paid  much 
attention  yet.  Although  we  expect  that  topological  insulators  remain  stable  at  weak  interactions, 
their  stability  under  intermediate  and  strong  interactions  is  an  important  open  question. 

Our  previous  work  of  Ref.  [43]  is  one  of  the  first  two  papers  explicitly  addressing  this  question 
(the  other  one  is  by  J.  Moore  et  al.  posted  at  the  same  time  [44]).  The  main  result  is  that  the  Z2  ar¬ 
gument  provided  by  Kane  and  Mele  [33]  needs  reconsideration  in  the  presence  of  interactions.  Let 
us  consider  the  ID  helical  edge  modes  of  the  2D  TIs,  although  the  single  particle  backscattering  is 
forbidden  by  TR  symmetry,  the  two-particle  correlated  backscattering  are  allowed  by  TR  symme¬ 
try.  The  two-particle  backscattering  is  an  interaction  effect.  Under  strong  repulsive  interactions,  its 
effects  are  non-perturbative  and  can  gap  out  the  edge  modes  by  spontaneously  breaking  TR  sym¬ 
metry.  In  other  words,  the  helical  edge  modes  become  unstable.  Concrete  criteria  for  the  stability 
of  the  helical  edge  modes  were  given  in  terms  of  the  phenomenological  Luttinger  parameters  based 
on  the  field  theoretical  analysis  of  bosonization. 

2.1  Sign  problem  free  QMC  simulation  to  the  Kane-Mele-Hubbard  model 

Based  on  our  past  analytical  work,  we  further  performed  the  numerical  simulations  of  quantum 
Monte  Carlo  (QMC)  for  the  quantitative  results  of  strong  interaction  effects  in  topological  insula¬ 
tors  [45].  Our  work  is  one  of  the  first  two  papers  using  QMC  to  study  the  interaction  effects  in 
topological  insulators  (The  other  one  is  by  Assaad’s  group  [46].  These  two  works  are  independent 
from  each  other). 

QMC  plays  an  important  role  in  the  study  of  strong  correlation  effects  of  fermions,  and  is  cur¬ 
rently  the  only  scalable  numerical  method  generating  exact  results  in  2D  under  suitable  conditions. 
However,  QMC  suffers  from  the  notorious  “sign  problem”  for  most  fermion  systems.  This  prob¬ 
lem  is  the  major  obstacle  for  QMC  to  apply  in  strongly  correlated  systems.  It  refers  to  the  fact 
that  the  amplitudes  of  a  quantum  process  of  fermions  are  usually  not  positive-definite,  thus  cannot 
be  directly  interpreted  as  classic  probability.  If  the  “sign  problem”  appears,  numerical  errors  grow 
exponentially  as  increasing  sample  size  and  decreasing  temperature  and  QMC  fails.  Under  certain 
situations,  for  example,  the  Hubbard  model  in  the  bipartite  lattice  at  half-filling,  the  sign  problem 
disappears  due  to  the  particle-hole  symmetry.  In  this  case,  conclusive  results  can  be  drawn  from  the 
high  precession  simulations.  A  celebrated  result  is  the  existence  of  the  antiferromagnetic  (AFM) 
long-range-order  at  zero  temperature  in  the  2D  square  lattice  [47]. 

Fortunately,  we  have  proved  that  for  the  prototype  model  for  the  2D  TIs,  the  Kane-Mele  model 
[33]  with  the  augmentation  of  the  Hubbard  interaction,  the  sign  problem  is  absent.  This  is  counter¬ 
intuitive:  the  free  part  of  the  Kane-Mele-Hubbard  model  involves  the  next-nearest-neighbor  (NNN) 
hopping  in  the  honeycomb  lattice.  Usually,  the  NNN  hopping  breaks  the  particle-hole  symmetry  of 
the  Hubbard  model  and  leads  to  the  sign  problem  and  the  failure  of  QMC.  Nevertheless,  in  the  case 
that  the  NNN  hopping  is  purely  imaginary,  the  Kane-Mele-Hubbard  model  is  in  fact  particle-hole 
symmetry  at  half-filling. 

Phase  diagram  of  the  Kane-Mele-Huhhard  model  using  QMC  Our  results  from  QMC  sim¬ 
ulations  are  depicted  in  Fig.  6  LHS.  U  is  the  Hubbard  interaction;  X  is  the  spin-orbit  coupling 
strength;  t  is  the  hopping  amplitude.  Clearly  as  U/t  goes  large,  the  system  finally  goes  into  long- 
range-ordered  antiferromagnetic  (AFM)  phase  whose  boundary  is  marked.  The  AFM  phase  is  a 
topologically  trivial  phase  whose  bulk  state  breaks  TR  symmetry.  There  are  no  non-trivial  gapless 
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Figure  6:  Left  hand  side  (LHS):  A  plot  of  the  phase  diagram  of  the  Kane-Mele-Hubbard  model  from  our 
QMC  simulations.  The  magnetic  phase  boundary  is  marked.  The  line  of  X  =  0  corresponds  to  the  case  of 
pure  Hubbard  model  in  the  honeycomb  lattice  that  was  simulated  in  Ref.  [48]  by  Meng  et  al. .  Right  hand 
side  (RHS):  The  two-point  equal-time  spin  correlation  function  {G\Sx{r)Sx{r  )|G)  along  the  zig-zag  edge 
with  X  =  0. 1  and  U  =  3.  The  length  of  the  zig-zag  edge  is  2  x  34  and  the  width  of  the  ribbon  is  4.  Because 
the  zig-zag  edge  contains  the  sites  of  both  A  and  B  sub-lattices,  the  fitted  decay  powers  are  slightly  different. 
We  use  their  average  value  to  obtain  the  Luttinger  parameter  K  «  0.21. 

edge  modes  in  the  AFM  phase.  Due  to  spin-orbit  coupling,  the  spin  SU  (2)  symmetry  is  explicitly 
broken.  The  AFM  moments  favor  lying  in  the  vy-plane,  thus  it  is  in  the  XT-class. 

There  is  a  large  portion  in  the  phase  diagram  at  small  and  intermediate  values  of  t//t  in  which 
the  system  remains  paramagnetic.  Apparently,  TR  symmetry  is  maintained  in  the  paramagnetic 
states.  At  U /t  =  0,  the  system  is  non-trivial  as  the  topological  band  insulator.  But  the  question 
is  that  whether  all  the  regime  of  the  paramagnetic  state  is  topologically  non-trivial.  Naively,  one 
would  expect  that  the  system  remains  non-trivial  with  gapless  edge  modes  until  the  bulk  becomes 
AFM  long-range-ordered.  Here  we  will  show  that  this  picture  is  incorrect. 

Destabilization  of  the  helical  edge  modes  A  major  result  of  our  QMC  simulations  is  that  the 
edges  of  the  Kane-Mele-Hubbard  systems  can  be  nearly  AFM  long-range-ordered  while  the  bulk 
remains  paramagnetic.  Because  of  the  band  gap  inside  the  bulk,  the  interaction  needs  to  be  stronger 
than  the  bulk  band  gap  to  drive  AFM.  Nevertheless,  edges  are  gapless  at  weak  interactions  due  to 
the  non-trivial  band  topology.  Interaction  effects  are  prominent  in  gapless  systems,  thus  the  AFM 
correlations  are  stronger  along  edges  than  in  the  bulk.  It  is  reasonable  to  expect  that  edges  develop 
AFM  earlier  while  the  bulk  remains  TR  invariant  and  paramagnetic. 

In  Fig.  6  RHS,  we  calculate  the  spin-spin  correlation  functions  along  the  edge,  which  shows 
the  power-law  correlations.  This  means  the  developing  of  the  quasi-long-range  order.  The  only 
reason  that  prevents  the  developing  of  true  long-rang  order  is  the  conservation  of  for  the  purpose 
to  remove  the  sign  problem.  However,  this  feature  is  not  required  for  the  most  general  models 
of  topological  insulators.  This  extra  f/(l)  symmetry  cannot  be  spontaneously  broken  in  ID  edge 
systems  due  to  a  rigorous  result  of  Winger-Mermin-Coleman  theorem.  Nevertheless,  in  the  Fig. 
6  RHS  with  an  intermediate  value  of  U /t  =  3,  the  average  decay  power  of  the  AFM  two-point 
function  is  very  small.  Actually,  from  our  previous  analytical  results,  as  long  as  the  condition  of 
S^-conservation  is  released,  the  edge  will  be  AFM  long-range-ordered  and  gapped. 

In  our  previous  bosonization  analysis,  the  decay  power  of  the  edge  spin  correlation  function 
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is  2K,  where  K  is  the  phenomenological  Luttinger  parameter  to  describe  the  helical  edge  modes. 
At  ^  the  two-particle  backscattering  term  mentioned  before  becomes  non-perturbative,  and 
destabilizes  the  helical  edge  modes  by  developing  the  true  long-range  ordered  AFM.  Such  a  term  in 
principle  is  allowed  to  exist  because  it  maintains  TR  symmetry.  Thus  we  can  divide  the  above  para¬ 
magnetic  regime  in  Fig.  6  LHS  into  two  parts  according  K  <\  andiiT  ,  respectively.  The  regime 
oiK  >2  corresponds  to  weak  interactions  in  which  the  edge  remains  stable  against  the  two-particle 
backscattering;  while  the  regime  of  K  <h  corresponds  to  intermediate  interactions,  in  which  the 
bulk  remains  paramagnetic  and  the  edge  is  unstable  under  the  two-particle  backscatterings.  A  pre¬ 
cise  phase  boundary  between  them  requires  more  detailed  numerical  work.  Nevertheless,  we  have 
shown  that  the  case  depicted  in  Fig.  6  already  belongs  to  the  intermediate  interaction  regime  and 
the  helical  edge  modes  are  unstable. 

2.2  Kondo  problem  in  the  helical  edge  modes 

Magnetic  impurities  have  dramatic  effects  in  transport  properties  known  as  Kondo  effect  [49,  50, 
51,  52,  53,  54,  55].  The  Kondo  effect  in  the  helical  edge  states  of  the  2D  topological  insulators  is 
an  important  question  both  theoretical  and  experimental. 

Initialed  in  a  previous  work  [56]  and  continued  in  a  recent  paper  [57]  in  collaboration  with  S. 
C.  Zhang’s  group,  we  have  performed  a  thorough  study  of  the  Kondo  effect  in  the  helical  edges 
of  topological  insulators.  We  have  performed  a  renormalization  group  analysis  with  new  features 
brought  by  interactions.  It  is  remarkable  that  the  repulsive  interactions  shift  the  critical  Kondo  cou¬ 
pling  constant  to  the  ferromagnetic  side,  which  means  that  Kondo  singlet  can  be  formed  with  both 
the  antiferromagnetic  and  weak  ferromagnetic  couplings.  After  the  formation  of  the  Kondo  singlet, 
it  behaves  like  a  spinless  impurity,  which  can  only  cause  phase  shift  to  the  edge  electrons.  We  fur¬ 
ther  calculated  the  edge  conductance  of  the  2D  topological  insulator  as  a  function  of  temperature 
in  the  presence  of  a  magnetic  impurity.  At  high  temperatures,  Kondo  and/or  two-particle  scatter¬ 
ing  give  rise  to  a  logarithmic  temperature  dependence.  At  low  temperatures,  for  weak  Coulomb 
interactions  in  the  edge  liquid  the  conductance  is  restored  to  unitarity  with  unusual  power-laws 
characteristic  of  a  ‘local  helical  liquid’,  while  for  strong  interactions  transport  proceeds  by  weak 
tunneling  through  the  impurity  where  only  half  an  electron  charge  is  transferred  in  each  tunneling 
event. 


3  Topological  insulators  from  3D  isotropic  Landau  levels 

Introduction  If  we  recall  the  study  of  topological  states  in  condensed  matter  physics,  it  starts 
from  the  2D  QHE  systems  with  Landau  levels  (LL).  Later  on,  topological  states  were  generalized 
to  systems  with  Bloch-wave  band  structures  but  not  LLs  [58],  which  were  known  as  quantum 
anomalous  Hall  insulators.  The  research  of  the  TR  invariant  topological  insulators  in  2D  and  3D 
mainly  focuses  on  Bloch-wave  band  structures  with  the  non-trivial  topological  Z2-index.  However, 
the  3D  TR  invariant  topological  insulators  based  on  LLs  have  not  been  studied  before.  Usually, 
LLs  are  based  on  the  planar  structure  subjected  to  the  external  magnetic  field,  thus  it  is  not  easy  to 
generalize  to  3D.  Nevertheless,  we  have  made  a  breakthrough  successfully  generating  LLs  to  3D 
with  the  full  rotational  symmetry  [59,  60]. 

Compared  to  the  usual  topological  insulators  with  Bloch-wave  band  structures,  LLs  have  ap¬ 
parent  advantages.  LLs  are  flat  without  dispersions,  thus  interaction  effects  are  non-perturbative. 
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Figure  7:  LHS)  The  eigenstates  of  the  3D  harmonic  oscillator  labeled  hy  total  angular  momentum  j±  = 
Z  ±  Following  the  solid  diagonal  (dashed)  lines,  these  states  are  reorganized  into  the  fully  rotational 
symmetric  3D  Landau  level  sates  with  the  positive  (negative)  helicity.  RHS)  Each  LL  contributes  one  branch 
of  helical  surface  mode.  The  energy  dispersion  of  the  first  four  Landau  levels  v.s.  7+  =  Z  +  5.  Open  boundary 
condition  is  used  for  a  ball  with  the  radius  Ro/lg  =  8. 


Furthermore  LLs  possess  beautiful  analytieal  properties  whieh  do  not  exhibit  in  other  band  strue- 
ture  topologieal  insulators.  In  faet,  the  study  of  the  2D  fraetional  QHE  effeets  is  greatly  benefited 
from  the  analytieal  properties  of  2D  LL.  It  is  reasonable  to  expeet  fractional  topological  states  in 
3D  with  LLs,  which  will  be  our  next  research  focus. 

3D  LLs  from  Aharanov-Casher  coupling  Our  mechanism  is  actually  very  simple.  Let  us  start 
from  the  2D  LL  problem  to  gain  some  intuition,  in  which  the  Z7(l)  magnetic  vector  potential  A 
minimally  couples  to  the  canonical  momentum.  In  the  symmetric  gauge,  A  =  xr.  A  nice 
observation  is  that  the  2D  LL  Hamiltonian  is  just  2D  harmonic  potential  plus  the  orbital  Zeeman 
coupling.  The  2D  LL  wavefunctions  are  just  those  of  the  2D  harmonic  oscillator.  The  harmonic 
oscillator  spectra  exhibit  the  dispersion  with  respect  to  the  orbital  angular  momentum  m  which  is 
exactly  canceled  by  the  opposite  dispersion  of  m  from  the  orbital  Zeeman  coupling.  As  a  result, 
each  LL  becomes  dispersionless  with  respect  to  m.  In  other  words,  a  different  organization  of 
the  same  set  of  harmonic  oscillator  wavefunctions,  which  are  originally  topologically  trivial,  gives 
rise  to  the  topologically  non-trivial  LLs.  However,  the  external  magnetic  field  specifies  a  unique 
direction,  thus  it  is  not  easy  to  generalize  to  3D. 

Instead,  for  the  3D  case,  we  employ  an  Aharanov-Casher  potential  with  the  replacement  of  B 
by  the  three  Pauli  matrices  a  as  A(xp  =  x  r,  which  is  essentially  an  SU (2)  gauge  potential. 
This  Hamiltonian  is  actually  the  3D  harmonic  oscillator  plus  spin-orbit  coupling.  The  spectra  of 
the  3D  harmonic  oscillators  is  depicted  in  Lig.  7  LHS  as  organized  in  the  total  angular  momentum 
eigen-basis  of  J  =  L  +  S.  Lor  each  orbital  angular  momentum  /,  there  are  two  different  states  with 
opposite  helicities  j±  —  I  Depending  on  the  sign  of  the  spin-orbit  coupling  term,  this  term 
turns  the  eigenstates  of  the  positive  or  negative  helicity  into  3D  flat  LLs.  In  particular,  the  3D  LLs 
also  have  a  nice  analytic  structure:  the  spin-orbit  coupled  spheric  harmonics  replaces  the  role  of 
azimuthal  angular  form  factor  for  the  2D  LLs. 
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Helical  surface  modes  -  3D  strong  topological  insulators  With  the  spherical  open  boundary 
condition,  we  find  that  each  LL  contributes  a  branch  of  helical  surface  mode.  The  calculated  surface 
spectra  are  depicted  in  RHS  of  Fig.  7.  At  small  values  of  I,  the  spectra  remain  flat  because  their 
wavefunctions  are  inside  the  bulk  and  thus  are  not  sensitive  to  the  boundary.  As  I  increases,  their 
wavefunctions  begin  to  touch  the  open  boundary,  and  thus  their  energy  is  pushed  upward.  Further 
analysis  shows  that  the  effective  edge  Hamiltonian  for  each  LL  is  actually  a  Dirac  Hamiltonian. 
Each  filled  LLs  contribute  one  branch  of  helical  Fermi  surface.  Thus  if  odd  number  of  LLs  are 
filled,  the  system  is  a  3D  strong  topological  insulator. 

F urther  developments  We  have  further  generalized  the  above  3D  LLs  for  non-relativistic  fermions 
to  Dirac  fermions,  which  can  be  considered  as  the  generalization  of  the  LLs  of  2D  graphene  sys¬ 
tems  to  the  3D  systems  with  the  full  rotational  symmetry.  In  fact,  we  have  also  generalized  both  of 
the  3D  non-relativistic  and  relativistic  LLs  into  arbitrary  dimensions.  Our  next  step  of  research  will 
focus  on  interaction  effects.  Our  dream  is  to  find  the  3D  fractional  topological  insulator  wavefunc¬ 
tions,  i.e.,  the  analogy  of  the  Laughlin  wavefunction  in  3D,  and  the  associated  exotic  many-body 
properties. 

4  Supercondutivity  in  iron-based  superconductors 

4.1  Time  reversal  (TR)  symmetry  breaking  pairing  states  in  iron-pnictide  superconductors 

The  discovery  of  the  iron-pnictide  superconductors  has  attracted  a  great  deal  of  attention  [61,  62, 
63,  64,  65,  66,  67].  They  are  novel  systems  with  unconventional  Cooper  pairing,  whose  pairing 
symmetry  remains  highly  debating.  This  problem  is  complicated  by  the  structure  of  their  multiple 
Fermi  surfaces,  which  include  the  hole  Fermi  surfaces  ai.2  around  the  F-point  at  (kx^ky)  —  (0,0), 
and  the  electron  Fermi  surfaces  |3i^2  around  theMy2  points  at  (71,0)  and  (0,7t),  respectively.  Many 
theoretical  proposals  have  suggested  the  fully-gapped  5±-wave  state  which  preserves  the  4-fold 
rotational  symmetry  [68,  69,  70,  71,  72].  Experimentally,  the  superfluid  density  obtained  from  the 
penetration  depth  measurements  is  insensitive  to  temperature,  consistent  with  this  picture  [73,  74, 
75].  Another  competing  pairing  structure  is  d^2_y2  as  proposed  by  several  groups  [69,  76,  40]. 

In  particular,  it  has  been  found  that  the  and  d^2_y2  pairings  are  nearly  degenerate  in  EaOEeP 
compounds  [77]. 

Our  proposal  -  time  reversal  symmetry  breaking  pairing  Considering  the  competing  nature 
between  s±  and  dx2_y2  pairings  in  iron-pnictide  superconductors,  we  propose  in  Ref.  [78]  that 
their  complex  mixture  4-  id j^2_y2  with  time  reversal  symmetry  breaking  can  appear  at  low  tem¬ 

peratures.  As  lowering  the  temperature,  two  consecutive  superconducting  transitions  occur.  The 
system  first  undergoes  the  transition  to  the  time  reversal  invariant  5±-wave  state,  and  then  into  the 
-1-  id^2_y2  state  by  breaking  time  reversal  symmetry.  We  also  predicted  various  experimentally 
testable  signatures,  including  spatial  inhomogeneity  induced  supercurrents,  and  a  novel  collective 
mode.  To  our  knowledge,  our  paper  is  the  first  one  to  study  time  reversal  symmetry  breaking 
Cooper  pairing  in  iron-pnictide  superconductors. 

Unconventional  superconductivity  with  time  reversal  symmetry  breaking  have  been  actively  in¬ 
vestigated  in  recent  years.  Previous  examples  include  the  Kerr  rotation  experiment  of  Sr2Ru04  con¬ 
sistent  with  the  p  +  ip  pairing  symmetry  [79],  and  the  neutron  scattering  experiments  of  YBa2Cu306+x 
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Figure  8:  Spatial  distributions  of  supercurrent  induced  by  spatial  inhomogeneity  in  the  +  id^2_y2 
pairing  state  of  iron-pnictide  superconductors.  LHS)  The  supercurrent  pattern  induced  by  a  single 
impurity  located  at  the  center.  RHS)  The  supercurrent  pattern  around  in  a  square  sample  with  open 
boundaries.  The  length  of  each  arrow  is  proportional  to  the  magnitude  of  the  supercurrent.  0  and 
0  indicate  the  vorticities  of  the  loop  currents. 

indicating  the  existence  of  the  loop  current  ordering  [80,  81,  82].  The  possible  time  reversal  sym¬ 
metry  breaking  pairing  state  in  iron-pnictide  systems  will  add  a  new  member  to  this  exotic  family 
if  it  is  testified  by  experiments. 

Spatial  inhomogeneity  induced  supercurrent  A  novel  experimentally  testable  consequence  of 
the  -|-  id^2_y2  pairing  is  that  spatial  inhomogeneity  induces  supercurrent.  In  contrast,  in  the  su¬ 
perconductor  with  either  5- wave  or  J-wave  pairing  which  preserves  time  reversal  symmetry,  spatial 
inhomogeneity  only  induces  the  variation  of  the  pairing  amplitude  without  affecting  the  pairing 
phase.  Thus  spatial  inhomogeneity  does  not  induce  supercurrent  simply  because  it  does  not  breaks 
time  reversal  symmetry. 

In  the  -|-  id^2_y2  pairing  state,  time  reversal  symmetry  is  already  broken.  In  the  homogeneous 
systems,  the  relative  phase  between  s±  and  d^2_y2  components  is  fixed  at  ±|,  thus  still  no  super¬ 
current  exits.  The  and  d^2_y2  components  can  be  coupled  by  the  following  gradient  term  in  the 
Ginzburg-Landau  free  energy  as 

ysd{K{^l-^y)^d+C.C.},  (2) 

which  is  allowed  by  all  the  symmetry  requirement.  In  spatially  inhomogeneous  systems,  the  varia¬ 
tion  of  the  order  parameters  cause  the  inhomogeneity  of  the  relative  phase  0^^  due  to  the  jsd  term, 
which  induces  supercurrent  even  in  the  absence  of  external  magnetic  field. 

We  numerically  solved  the  supercurrent  patterns  for  both  cases  of  a  single  impurity  and  a  square 
sample  with  open  boundaries.  The  results  are  depicted  in  Fig.  8  A  and  B,  respectively.  For 
the  single  impurity  case,  the  magnitude  of  the  supercurrent  rapidly  decays  beyond  the  order  of 
the  healing  length.  The  suppercurrent  pattern  is  symmetric  under  a  combined  operation  of  time 
reversal  and  the  rotation  of  ±90°.  Along  the  x  and  y-axis  passing  the  impurity,  the  supercurrents 
flow  along  these  axis  and  exhibit  the  pattern  of  “two  in”  and  “two  out”,  which  is  consistent  with  the 
reflection  symmetry  respect  to  the  x  and  y-axis  and  the  continuity  condition.  For  the  square  sample, 
the  continuity  condition  suppresses  the  supercurrent  except  at  the  four  comers  as  depicted  in  Fig. 
8  B.  Each  of  four  corners  develop  a  circulating  supercurrent  loop  whose  chirality  are  staggered  as 
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we  move  around  the  edges.  This  is  also  consistent  with  the  combined  symmetry  operation  of  time 
reversal  and  the  rotation  of  90°.  This  circulating  supercurrent  can  be  detected  by  using  the  scanning 
SQUID. 

Prediction  of  a  novel  collective  resonance  mode  In  the  case  that  the  -pairing  state  wins  over 
d^2_y2  which  occurs  in  many  compounds  in  the  iron-pnictide  family,  we  predict  a  novel  collective 
resonance  mode.  This  mode  excites  the  d^2_yi  pairing  in  the  background  of  and  is  described  by 

the  nematic  operator  It  lies  in  the  particle-hole  channel  with 

the  d^2_y2  symmetry,  which  is  precisely  the  electron  excitations  measured  in  the  Big  Raman  mode. 
Because  of  the  competing  nature  of  the  s±  and  djfi_y2  pairings,  this  mode  is  a  sharp  low  energy 
excitation  below  the  superconducting  gap.  It  is  not  damped  by  the  quasiparticle  excitations.  A 
successful  observation  of  such  mode  will  be  a  demonstration  of  the  completion  between  s±  and 
d^2_y2  pairings. 

4.2  Orbital  ordering  the  two-fold  pairing  symmetry  in  FeSe 

One  of  the  most  important  problem  in  the  iron-based  superconductors  is  the  nature  of  the  pairing 
symmetry.  For  most  materials  in  the  111  1 -family,  various  experimental  results  agree  with  the  node¬ 
less  extended  5±-wave  pairing.  However,  in  other  families  of  1 1 1  and  11,  whether  the  pairing  gap  is 
nodal  or  nodeless  is  still  controversial.  On  the  other  hand,  many  iron-based  superconductors  exhibit 
spontaneous  nematic  ordering  which  reduces  the  symmetry  from  the  4-fold  rotational  symmetry  to 
2-fold.  A  common  reason  is  the  developing  of  stripe-like  antiferromagnetism  whose  wavevector  is 
(7t,0)  or  (0,7r). 

In  collaboration  with  Q.  K.  Xue’s  experimental  group  at  Tsinghua  University  in  Beijing,  my 
group  helped  them  analyze  the  data,  which  yields  important  information  on  the  pairing  symmetry 
in  the  one  of  the  simplest  iron-based  superconductors  FeSe  [83].  Using  the  methods  of  scanning 
tunneling  microscopy  and  spectroscopy,  their  data  reveal  clear  evidence  for  a  gap  function  with 
nodal  points.  Furthermore,  they  also  measured  the  vortex  core  tunneling  spectra  under  external 
magnetic  fields.  Amazingly  the  vortex  core  are  highly  anisotropic  breaking  the  4-fold  rotational 
symmetry.  This  anisotropy  also  exhibits  in  the  real  space  imaging  of  the  resonance  scattering 
spectra  around  impurity  sites,  and  in  the  quasi-particle  interference  pattern.  Their  paper  is  recently 
published  in  Science. 

Different  from  previous  iron-based  materials  in  which  the  breaking  of  the  4-fold  rotational 
symmetry  to  2-fold  is  due  to  the  striped  antiferromagnetism.  It  is  known  that  at  ambient  pressure, 
there  is  no  antiferromagnetic  instability  in  FeSe.  In  considering  this  fact,  my  group  proposed  that 
the  anisotropy  is  due  to  orbital  ordering  -  orbital-dependent  reconstruction  of  electronic  structure 
in  FeSe.  The  apparent  nodal  structure  in  the  gap  function  can  be  understood  as  the  mixture  between 
two  different  types  of  extended  5- wave  pairings  including  the  nearest  neighbor  and  the  next-nearest 
neighbor  pairings.  If  the  nearest  neighbor  pairing  dominates,  the  gap  function  is  nodal.  In  FeSe, 
the  orbital  ordering  also  brings  anisotropy  to  the  gap  functions.  My  group  has  also  performed  the 
theory  calculation,  which  is  in  nice  agreement  with  Prof.  Xue’s  experimental  data  [84]. 


5  Summary  of  the  most  important  results 

We  summarize  the  most  significant  results  under  the  support  of  ARO  as  follows. 
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1.  We  constructed  the  microscopic  theory  for  the  unconventional  meta- magnetic  states  observed 
in  Sr3Ru207  systems  based  on  the  quasi- ID  band  structures  of  dxz  and  dyz  [3],  calculated  the 
quasi-particle  interference  pattern  measured  by  J.  C.  Davis’  group  [5,  27],  and  investigated 
the  thermodyanica  quantities  in  such  a  system  [6].  These  results  are  published  in  Ref.  [3,  5, 
6,  27], 

2.  We  studied  interaction  effects  in  topological  insulators,  including  the  quantum  Monte  Carlo 
(QMC)  simulation  of  the  phase  diagram  of  the  Kane-Mele-Hubbard  model  [45],  and  the 
Kondo  effect  in  the  helical  edge  states  of  the  topological  insulators  [57].  We  are  one  of  the 
first  two  groups  to  performing  the  QMC  calculation  to  the  interacting  topological  insulators. 
The  STM  spectra  in  the  helical  surface  states  of  Bi2Te3  are  calculated  in  a  good  agreement 
with  experiments  [85].  We  also  generalize  the  usual  Landau  level  to  three  dimensions,  which 
is  a  3D  topological  insulators  with  flat  bands  [59,  60].  These  results  are  published  in  Ref. 
[57,45,59,  60]. 

3.  We  have  also  investigated  the  unconventional  supercondutivity  in  iron-based  superconduc¬ 
tors.  We  are  the  first  group  to  propose  the  time-reversal  symmetry  breaking  pairing  states 
in  such  a  systems  [78].  We  also  help  the  experimentalist  to  analyze  their  STM  data  in  FeSe 
superconductor  and  find  the  evidence  of  orbital  ordering  [83].  A  theory  paper  is  also  written 
to  give  a  detailed  explanation  of  the  experiment  [84]. 
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